Non-transitory computer-readable storage medium, embroidery pattern displaying device, and method

ABSTRACT

A non-transitory computer-readable medium stores computer-readable instructions. The instructions, when executed, cause a processor of an embroidery pattern displaying device provided with a capturing portion and a displaying portion to perform steps. The steps include detecting a marker associated with an embroidery pattern arranged within an augmented reality space from a real image that is an image of a real space captured by the capturing portion. The steps include displaying, when the marker is detected from the real image, a pattern image indicating the embroidery pattern associated with the detected marker, superimposed on the real image with the detected marker as a reference, on the displaying portion.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of U.S. Ser. No.15/980,317, filed on May 15, 2018, which is a bypass continuation ofInternational Application No. PCT/JP2016/075798, filed Sep. 2, 2016,which claims priority from Japanese Patent Application No. 2015-232182,filed on Nov. 27, 2015. The disclosure of the foregoing application ishereby incorporated by reference in its entirety.

BACKGROUND

The present disclosure relates to a non-transitory computer-readablestorage medium, an embroidery pattern displaying device, and a method.

In the past, as augmented reality (AR) technology, there is known animage processing device that displays a virtual object corresponding toan image of a marker in a captured image. The above described imageprocessing device captures an image of a planned place of constructionof a building and obtains a landscape image. The image processing deviceidentifies a marker in the landscape image. The image processing devicealigns a building image with the landscape image, with the identifiedmarker as a reference, and displays the building image superimposed onthe landscape image.

SUMMARY

Technology for determining the arrangement of an embroidery pattern withrespect to a sewing workpiece using AR technology has not beenthoroughly studied.

Various embodiments of the broad principles derived herein provide anon-transitory computer-readable storage medium, an embroidery patterndisplaying device, and a method that make it possible to ascertain afinished image of an embroidery pattern when the embroidery pattern isarranged with respect to a sewing workpiece using AR technology.

Embodiments herein provide a non-transitory computer-readable mediumstoring computer-readable instructions. The instructions, when executed,cause a processor of an embroidery pattern displaying device providedwith a capturing portion and a displaying portion to perform steps thatinclude detecting a marker associated with an embroidery patternarranged within an augmented reality space from a real image that is animage of a real space captured by the capturing portion, and displaying,when the marker is detected from the real image, a pattern imageindicating the embroidery pattern associated with the detected marker,superimposed on the real image with the detected marker as a reference,on the displaying portion.

Embodiments herein also provide an embroidery pattern displaying deviceincludes a capturing portion, a displaying portion, a processor, and amemory. The capturing portion is configured to capture a real image thatis an image of a real space of an object. The displaying portion isconfigured to display the real image captured by the capturing portion,and a pattern image indicating an embroidery pattern. The memory storescomputer-readable instructions that, when executed by the processor,cause the embroidery pattern displaying device to detect a markerassociated with the embroidery pattern arranged within an augmentedreality space, from the real image captured by the capturing portion,and display, when the marker is detected, the pattern image indicatingthe embroidery pattern associated with the detected marker, superimposedon the real image captured by the capturing portion with the detectedmarker as a reference, on the displaying portion.

Embodiments herein also provide a method of controlling an embroiderypattern displaying device provided with a capturing portion and adisplaying portion. The method includes detecting a marker associatedwith the embroidery pattern arranged within an augmented reality space,from a real image that is an image of a real space captured by thecapturing portion. The method includes displaying, when the marker isdetected, the pattern image indicating the embroidery pattern associatedwith the detected marker, superimposed on the real image captured by thecapturing portion with the detected marker as a reference, on thedisplaying portion.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will be described below in detail with reference to theaccompanying drawings in which:

FIG. 1 is a perspective view of a sewing machine and an embroiderypattern displaying device;

FIG. 2 is an explanatory view illustrating a configuration of a lowerend portion of a head portion;

FIG. 3 is a block diagram illustrating an electrical configuration ofthe sewing machine and the embroidery pattern displaying device;

FIG. 4 is an explanatory view of a marker storage area, an embroiderydata storage area, and a sewing area storage area of flash memory;

FIG. 5 is a flowchart of main processing performed by the embroiderypattern displaying device;

FIG. 6 is an explanatory view of a real image;

FIG. 7 is an explanatory view of the real image, a pattern image, and anarea image displayed on a displaying portion of the embroidery patterndisplaying device;

FIG. 8 is an explanatory view of screens displayed on the displayingportion according to editing by operations (1) to (5) and editingcontent;

FIG. 9 is an explanatory view of a screen displayed on the displayingportion of the embroidery pattern displaying device; and

FIG. 10 is a flowchart of main processing performed by the sewingmachine.

DETAILED DESCRIPTION

Embodiments of the present disclosure will be described with referenceto the drawings. The physical configuration of a sewing system 100 thatincludes a sewing machine 1 and an embroidery pattern displaying device3 (hereinafter, simply referred to as “device 3”) will be described withreference to FIG. 1 and FIG. 2. The up and down direction, lower rightside, upper left side, lower left side, and upper right side in FIG. 1are the up and down direction, to the front, to the rear, to the left,and to the right, respectively, with respect to the sewing machine 1 andthe device 3. That is, the surface on which a liquid crystal display(hereinafter, simply referred to as LCD) 15, described later, isarranged is a front surface of the sewing machine 1. The longitudinaldirection of a bed portion 11 and an arm portion 13 is the left-rightdirection of the sewing machine 1. The side on which a pillar portion 12is arranged is the right side. The extending direction of the pillarportion 12 is the up and down direction of the sewing machine 1. In thepresent specification, image data to be processed by a computer is alsoreferred to simply as “image.”

[1. Physical configuration of sewing machine 1 and embroidery frame 50]

As illustrated in FIG. 1, the sewing machine 1 has the bed portion 11,the pillar portion 12, the arm portion 13, and a head portion 14. Thebed portion 11 is a base portion of the sewing machine 1 that extends inthe left-right direction. The pillar portion 12 stands upright,extending upward from the right end portion of the bed portion 11. Thearm portion 13 extends toward the left from the upper end of the pillarportion 12 so as to face the bed portion 11. The head portion 14 is aportion that connects to the left distal end portion of the arm portion13.

The bed portion 11 includes a needle plate 21 (refer to FIG. 2) on theupper surface of the bed portion 11. The needle plate 21 has a needlehole (not shown in the drawings) through which a sewing needle 7,described later, is able to be inserted. A sewing workpiece (e.g., workcloth), not shown, is placed on the upper surface of the needle plate21. The sewing machine 1 includes a feed dog, a feed mechanism, and ashuttle mechanism and the like, not shown, on the lower side of theneedle plate 21, i.e., inside the bed portion 11. The feed dog is drivenby the feed mechanism during normal sewing, which is not embroiderysewing, and moves the sewing workpiece a predetermined movement amount.The shuttle mechanism twists an upper thread (not shown in the drawings)with a lower thread (not shown in the drawings) below the needle plate21.

The sewing machine 1 includes a movement mechanism 40. The movementmechanism 40 is configured to be able to relatively move a sewingworkpiece C (hereinafter, also referred to as “object”) held by anembroidery frame 50 with respect to an image sensor 35 and a needle bar6 that will be described later. The movement mechanism 40 includes acarriage 42 and a main body portion 41. The carriage 42 includes a frameholder (not shown in the drawings), a Y-axis movement mechanism (notshown in the drawings), and a Y-axis motor 84 (refer to FIG. 3). Theframe holder is provided on the right side surface of the carriage 42.One embroidery frame 50 selected from among a plurality of types ofembroidery frames 50 that differ in shape and size can be attached toand detached from the frame holder. The Y-axis movement mechanism movesthe frame holder in the front-rear direction (Y-axis direction). TheY-axis motor 84 drives the Y-axis movement mechanism.

The embroidery frame 50 of the present embodiment has a first framemember 51, a second frame member 52, and an attachment portion 55, andis able to hold the sewing workpiece C with the first frame member 51and the second frame member 52. A sewing area 54 set within theembroidery frame 50 is a region where the sewing machine 1 can formstitches. The attachment portion 55 is a portion that attaches to theframe holder. The attachment portion 55 has a shape unique to theembroidery frame 50. When the embroidery frame 50 is attached to theframe holder of the movement mechanism 40, the sewing machine 1 canidentify that the embroidery frame 50 has been attached as well asidentify the type of the embroidery frame 50, on the basis of the uniqueshape of the attachment portion 55 detected by a detector 36 (refer toFIG. 3) that will be described later.

The main body portion 41 includes an X-axis movement mechanism (notshown in the drawings) and an X-axis motor 83 (refer to FIG. 3) insidethe main body portion 41. The X-axis movement mechanism moves thecarriage 42 in the left-right direction (X-axis direction). The X-axismotor 83 drives the X-axis movement mechanism. The movement mechanism 40can move the embroidery frame 50 that is attached to the carriage 42(more specifically, the frame holder) to a position indicated by aparticular XY coordinate system (embroidery coordinate system). In theembroidery coordinate system, directions to the right, left, forward,and rearward with respect to the sewing machine 1 are an X plusdirection, an X minus direction, a Y minus direction, and a Y plusdirection, for example.

The LCD 15 is provided on the front surface of the pillar portion 12. Animage that includes a variety of items such as commands, illustrations,setting values, messages, and the like is displayed on the LCD 15. Atouch panel 26 capable of detecting a pressed position is provided onthe front surface side of the LCD 15. When a user performs a pressingoperation with respect to the touch panel 26 using a finger or a styluspen (not shown in the drawings), the pressed position is detected by thetouch panel 26. A CPU 61 (refer to FIG. 3) of the sewing machine 1detects a selected item in the image, on the basis of the detectedpressed position. Hereinafter, the pressing operation with respect tothe touch panel 26 by the user will be referred to as a panel operation.The user can select a pattern to be sewn and a command to be executedand the like by a panel operation. The pillar portion 12 includes asewing machine motor 81 (refer to FIG. 3) inside the pillar portion 12.

A cover 16 that can be opened and closed is provided on an upper portionof the arm portion 13. FIG. 1 illustrates a state in which the cover 16is open. A thread storage portion 18 is provided below the cover 16(that is, inside the arm portion 13). The thread storage portion 18 canstore a thread spool 20 around which the upper thread is wound. A mainshaft (not shown in the drawings) that extends in the left-rightdirection is provided inside the arm portion 13. The main shaft isrotatably driven by the sewing machine motor 81. Various switchesincluding a start/stop switch 29 are provided on a lower left portion ofthe front surface of the arm portion 13. The start/stop switch 29 startsand stops operation of the sewing machine 1. That is, the start/stopswitch 29 is used to input commands to start and stop sewing.

As illustrated in FIG. 2, the needle bar 6, a presser bar 8, and aneedle bar up-and-down movement mechanism 34 and the like are providedat the head portion 14. The sewing needle 7 is detachably attached tothe lower end of the needle bar 6. A presser foot 9 is detachablyattached to the lower end portion of the presser bar 8. The needle bar 6is provided at the lower end of the needle bar up-and-down movementmechanism 34. The needle bar up-and-down movement mechanism 34 drivesthe needle bar 6 in the up and down direction by the rotation of themain shaft.

The image sensor 35 having a predetermined capture area is providedinside the head portion 14. The image sensor 35 is a well-knownComplementary Metal Oxide Semiconductor (CMOS) image sensor, forexample. The image sensor 35 is a publicly known area sensor in which aplurality of pickup elements 37 lined up in a main scanning direction(for example, a plurality of CMOS), and are arranged in a plurality ofrows in a sub-scanning direction. In the present embodiment, the mainscanning direction corresponds to the X-axis direction (left-rightdirection) of the sewing machine 1, and the sub-scanning directioncorresponds to the Y-axis direction (the front-rear direction).

The image sensor 35 is arranged so as to be able to capture a regionthat includes an area below the needle bar 6, and can output image data.The outputted image data is stored in a predetermined storage area of aRAM 63 (refer to FIG. 3). A coordinate system of the image indicated bythe image data outputted from the image sensor 35 (hereinafter, alsoreferred to as “image coordinate system”) and a coordinate system of theentire space (hereinafter, also referred to as “world coordinatesystem”) are associated beforehand by a parameter stored in a flashmemory 64 (refer to FIG. 3). The world coordinate system and theembroidery coordinate system are associated beforehand by a parameterstored in the flash memory 64. Therefore, the sewing machine 1 canperform processing that identifies the coordinates in the embroiderycoordinate system on the basis of the image data outputted from theimage sensor 35.

[2. Physical Configuration of Device 3]

The physical configuration of the device 3 will be described withreference to FIG. 1. The device 3 is a well-known tablet-type personalcomputer (PC). The device 3 includes an operating switch 131, adisplaying portion 135, and a touch panel 132 on the upper surface ofthe device 3. The operating switch 131 is used when inputting variouscommands into the device 3. An image that includes various items such ascommands, illustrations, setting values messages, and the like isdisplayed on the displaying portion 135. The touch panel 132 is providedon the front side surface of the displaying portion 135, and can detecta pressed position. When the user performs a pressing operation withrespect to the touch panel 132 using a finger or a special touch pen,the pressed position is detected by the touch panel 132. A selected itemin the image is detected on the basis of the detected pressed position.The device 3 includes an image sensor 134 (refer to FIG. 3) on a bottomsurface. The image sensor 134 is a well-known CMOS image sensor, forexample.

[3. Electrical Configuration of Sewing Machine 1]

The electrical configurations of the sewing machine 1 and the device 3of the sewing system 100 will be explained in order with reference toFIG. 3. As illustrated in FIG. 3, the sewing machine 1 includes the CPU61, a ROM 62, the RAM 63, the flash memory 64, an input/output interface(I/O) 66, and a communication I/F 67. The CPU 61 is connected to the ROM62, the RAM 63, the flash memory 64, the input/output I/O 66, and thecommunication I/F 67 via a bus 65.

The CPU 61 is responsible for the main control of the sewing machine 1,and performs various calculations and processing related to capturingand sewing, in accordance with various programs stored in the ROM 62.Although not shown, the ROM 62 includes a plurality of storage areasincluding a program storage area. Various programs for operating thesewing machine 1 (for example, a program for performing main processingthat will be described later) are stored in the program storage area.

A storage area within which is stored calculation results and the likecalculated by the CPU 61 is provided in the RAM 63. The flash memory 64includes a plurality of storage areas in which are stored variousparameters and the like for the sewing machine 1 to perform variousprocessing. The plurality of storage areas of the flash memory 64include a marker storage area 68, an embroidery data storage area 69,and a sewing area storage area 70, which will be described later withreference to FIG. 4. Drive circuits 71 to 74, the touch panel 26, thestart/stop switch 29, the image sensor 35, and the detector 36 areconnected to the input/output I/O 66. The detector 36 is configured todetect that the embroidery frame 50 is attached to the movementmechanism 40, and output a detection result that corresponds to the typeof the embroidery frame 50.

The sewing machine motor 81 is connected to the drive circuit 71. Thedrive circuit 71 drives the sewing machine motor 81 in accordance with acontrol signal from the CPU 61. The needle bar up-and-down movementmechanism 34 (refer to FIG. 2) is driven via the main shaft (not shownin the drawings) of the sewing machine 1 with the driving of the sewingmachine motor 81, such that the needle bar 6 moves up and down. TheX-axis motor 83 is connected to the drive circuit 72. The Y-axis motor84 is connected to the drive circuit 73. The drive circuits 72 and 73drive the X-axis motor 83 and the Y-axis motor 84, respectively, inaccordance with control signals from the CPU 61. The embroidery frame 50that is attached to the movement mechanism 40 moves in the left-rightdirection (X-axis direction) and the front-rear direction (Y-axisdirection) by a movement amount corresponding to the control signals,with the driving of the X-axis motor 83 and the Y-axis motor 84. Thedrive circuit 74 displays an image on the LCD 15 by driving the LCD 15in accordance with a control signal from the CPU 61. The communicationI/F 67 is an interface for connecting the sewing machine 1 to a network10. The CPU 61 can send and receive data to and from other devices (thedevice 3, for example) that connect to the network 10, via thecommunication I/F 67.

The operation of the sewing machine 1 will now be briefly described. Atthe time of embroidery sewing using the embroidery frame 50, theembroidery frame 50 is moved in the X-axis direction and the Y-axisdirection by the movement mechanism 40, and the needle bar up-and-downmovement mechanism 34 (refer to FIG. 2) and the shuttle mechanism (notshown in the drawings) are driven. As a result, an embroidery pattern issewn onto the sewing workpiece C held by the embroidery frame 50, by thesewing needle 7 that is attached to the needle bar 6.

[4. Electrical Configuration of Device 3]

As illustrated in FIG. 3, the device 3 includes a CPU 121, a ROM 122, aRAM 123, a flash memory 124, a communication I/F 125, and aninput/output interface 128. The CPU 121 is responsible for controllingthe device 3. The CPU 121 is electrically connected to the ROM 122, theRAM 123, the flash memory 124, the communication I/F 125, and theinput/output interface 128 via a bus 127. A boot program and BIOS andthe like are stored in the ROM 122. Temporary data is stored in the RAM123. Various setting values, including a marker storage area 137 and asewing area storage area 138, which will be described later withreference to FIG. 4, are stored in the flash memory 124. Thecommunication I/F 125 is an interface for connecting the device 3 to thenetwork 10. The CPU 121 can send and receive data to and from otherdevices (the sewing machine 1, for example) that connect to the network10, via the communication I/F 125.

The input/output interface 128 is connected to the operating switch 131,the touch panel 132, the image sensor 134, and the displaying portion135. The image sensor 134 outputs image data obtained by capturing apredetermined capture area. The outputted image data is stored in theRAM 123. The displaying portion 135 is a liquid crystal display.

[5. Storage Area of Flash Memory 64 and 124]

The marker storage area 68, the embroidery data storage area 69, and thesewing area storage area 70 will now be described in order withreference to FIG. 4. The marker storage area 68, the embroidery datastorage area 69 and the sewing area storage area 70 are provided in theflash memory 64 of the sewing machine 1. The marker storage area 68stores data indicating a marker 4 used when embroidery pattern editingprocessing is performed by the sewing system 100. The data indicatingthe marker 4 is used in processing that extracts the marker 4 from theimages obtained from the image sensors 35 and 134. The data indicatingthe marker 4 may be image data indicating the marker 4, or numericaldata indicating the size and shape of the marker 4, for example.

The marker 4 serves as a reference for the size and arrangement of anembroidery pattern 5 when editing an embroidery pattern to be sewn bythe device 3. The marker 4 further serves as a reference when arrangingthe embroidery pattern based on the marker 4 in accordance with theediting content in the device 3, in the sewing machine 1. The marker 4is associated with the embroidery pattern to be sewn. Therefore, themarker 4 may be any marker that is captured by both the device 3 and thesewing machine 1, and from which the size and position of the marker 4can be detected from the obtained image. The marker 4 includes a whitethin sheet-like sheet, and a line-drawing that is drawn in black on thesurface of the sheet. A transparent adhesive is applied to the backsurface of the sheet. The user is able to adhere the sheet onto thesewing workpiece C. The line-drawing drawn on the upper surface of thesheet includes a character pattern 47 and a graphic pattern 48. Thecharacter pattern 47 is a pattern representing an uppercase letter K.The graphic pattern 48 is a pattern representing a square. The dataindicating the marker 4 is image data indicating the surface of themarker 4. The left-right direction and the up and down direction in FIG.4 correspond to the X direction and the Y direction in the embroiderycoordinate system, respectively. The flash memory 124 of the device 3has the marker storage area 137 and stores data indicating the marker 4,similar to the marker storage area 68.

The embroidery data storage area 69 stores embroidery data, which isdata indicating an embroidery pattern, for each of a plurality ofembroidery patterns. The embroidery data includes an ID, sewing data,and pattern image data. The ID is a unique identifier given to each ofthe plurality of embroidery patterns. The ID is a character string inwhich letters and numbers are combined. The sewing data includescoordinate data for each thread color data. The thread color data isdata indicating the color of thread forming a stitch. The coordinatedata is data indicating a stitch forming position (needle drop position)included in the embroidery pattern by coordinates in the embroiderycoordinate system. That is, the sewing data includes data indicating thestitch forming position for each stitch color. The pattern image data isdata that indicates a pattern image 46 indicating the finishedembroidery pattern. When sewing an embroidery pattern using the sewingmachine 1, the user selects a desired embroidery pattern from aplurality of embroidery patterns on the basis of a plurality ofembroidery data stored in the flash memory 64. The embroidery data maybe obtained from a storage device such as an external device that isconnected via the communication I/F 67 or a memory card that iselectrically connected to the sewing machine 1.

The embroidery pattern 5 is an embroidery pattern to be sewn with asingle color thread, which represents an uppercase letter A. Theleft-right direction in FIG. 4 corresponds to the X direction in theembroidery coordinate system, and the up and down direction correspondsto the Y direction. The size of the embroidery pattern 5 is indicated bythe size of the smallest rectangle 56 containing the embroidery pattern5. In this example, the embroidery pattern 5 is associated with themarker 4. More specifically, with the embroidery pattern 5, the initialsize and arrangement of the embroidery pattern 5 with respect to themarker 4 are determined as shown in FIG. 4. The initial size of theembroidery pattern 5 with respect to the marker 4 is determined suchthat the size of the embroidery pattern 5 in the Y direction is 1.7times the size of the marker 4 in the Y direction. The initialarrangement of the embroidery pattern 5 with respect to the marker 4 isdetermined such that the center of the marker 4 is aligned with thecenter of the embroidery pattern 5. The initial size and arrangement ofthe embroidery pattern 5 with respect to the marker 4 may be differentor the same for each embroidery pattern. The initial size andarrangement of the embroidery pattern 5 with respect to the marker 4 mayalso be able to be set by the user.

The sewing area storage area 70 stores the ID indicating the type of theembroidery frame 50 and the size of the sewing area with respect to themarker 4 in association with each other. The ID is a character string inwhich letters and numbers are combined. The left-right direction in FIG.4 corresponds to the X direction in the embroidery coordinate system,and the up and down direction corresponds to the Y direction. The sewingarea is a rectangular region that extends in the X direction and the Ydirection. The sewing area storage area 70 also stores the position ofthe sewing area with respect to the embroidery frame 50. The flashmemory 124 of the device 3 has the sewing area storage area 138, andstores the ID indicating the type of the embroidery frame 50 and thesize of the sewing area with respect to the marker 4 in association witheach other, similar to the sewing area storage area 70. The flash memory124 also stores the initial arrangement of the sewing area with respectto the marker 4. The size and arrangement of the sewing area 54 withrespect to the graphic pattern 48 of the marker 4 in the embroideryframe 50 with an ID of W001 differ from the size and arrangement of thesewing area 54 with respect to the graphic pattern 48 of the marker 4 inthe embroidery frame 50 with an ID of W002. In this example, the sizeand arrangement of the sewing area with respect to the marker 4 differfor each type of embroidery frame 50.

[6. Outline of Embroidery Pattern Editing Processing]

The outline of embroidery pattern editing processing able to beperformed by the sewing system 100 will now be described. With thesewing system 100, the arrangement of the embroidery pattern sewn by thesewing machine 1 can be edited by the device 3. When the embroiderypattern to be sewn and the type of the embroidery frame 50 to be used atthe time of sewing are selected, the sewing machine 1 sends dataindicating the selected embroidery pattern and data indicating aneditable area to the device 3. The editable area is an area within whichthe embroidery pattern 5 indicated by the data received in step S1(refer to FIG. 4) can be edited in the device 3. The editable area is anarea of the same size and shape as the sewing area. The device 3receives data indicating the embroidery pattern sent from the sewingmachine 1. The user adheres the marker 4 to the sewing workpiece C(object) on which the embroidery pattern is to be sewn, and uses thedevice 3 to capture a real space that includes the marker 4 that willserve as a reference for the arrangement of the embroidery pattern. Thedevice 3 detects the marker 4 from a real image in which of the realspace is captured, and displays a pattern image 46 indicating theembroidery pattern on the basis of the received data, superimposed on anarea image indicating the editable area with the detected marker 4 asthe reference, on the displaying portion 135. The user edits theembroidery pattern while referring to the image displayed on thedisplaying portion 135. When the device 3 receives a command to endediting of the embroidery pattern, the device 3 sends editing data tothe sewing machine 1 via the communication I/F 125. The editing data isdata indicating the editing content of the embroidery pattern. Whenediting of the embroidery pattern in the device 3 is complete, the usercauses the embroidery frame 50 that has been selected to be used to holdthe sewing workpiece C to which the marker 4 has been adhered, withoutthe marker 4 being peeled off. The user then attaches the embroideryframe 50 holding the sewing workpiece C to the sewing machine 1. Thesewing machine 1 captures the sewing workpiece C that is held to theembroidery frame 50 and on which the marker 4 is arranged, and detectsthe marker 4 from the outputted image data. The sewing machine 1corrects embroidery data (sewing data) of the embroidery pattern on thebasis of the editing data received from the device 3 with the detectedmarker 4 as a reference. The sewing machine 1 sews the embroiderypattern in accordance with the corrected embroidery data.

[7. Processing Performed by Device 3]

The main processing of the device 3 will be described with reference toFIG. 5 to FIG. 9. In the main processing of the device 3, processingthat edits the embroidery pattern 5 selected by the sewing machine 1 andconfirming the editing content is performed using AR technology. In themain processing of the device 3, processing that sends the editing datato the sewing machine 1 is further performed. The main processing isperformed when the user operates the device 3 and inputs a command tolaunch an application to perform the main processing. When the commandis detected, the CPU 121 of the device 3 reads an embroidery patternediting program for performing the main processing stored in the programstorage area of the ROM 122 to the RAM 123. The CPU 121 performs thefollowing steps according to commands included in the embroidery patternediting program read to the RAM 123. Various parameters necessary forperforming the main processing are stored in the flash memory 124.Various data obtained in the main processing is stored in the RAM 123 asappropriate. Hereinafter, in order to simplify the description, a casewill be explained in which the embroidery pattern 5 to be sewn isselected and the embroidery frame 50 to be used when sewing theembroidery pattern 5 is selected, in the sewing machine 1 before themain processing starts.

In the main processing illustrated in FIG. 5, the CPU 121 receives an IDof the embroidery frame 50 data indicating the embroidery pattern 5associated with the marker 4 sent from the sewing machine 1, and dataindicating the editable area (step S1). The data indicating theembroidery pattern 5 is pattern image data of the embroidery pattern 5selected as the pattern to be sewn by the sewing machine 1. The dataindicating the editable area is data indicating the editable area to beused in sewing the embroidery pattern. The ID of the embroidery frame 50is information identifying the type of the embroidery frame 50. Dataindicating the size and shape of the editable area may be obtained asthe data indicating the editable area.

The CPU 121 outputs a command to the image sensor 134 to start capturing(step S2). The image sensor 134 starts processing that captures an imageand outputs the image data, on the basis of the command from the CPU121. The CPU 121 obtains a real image 43 indicated by the image dataoutputted from the image sensor 134 (step S3). The CPU 121 obtains thereal image 43 in FIG. 6, for example. As illustrated in FIG. 6, the realimage 43 is an image showing the manner in which the marker 4 is adheredto an area on the left chest portion of a white T-shirt 2.

The CPU 121 image-processes the real image 43 obtained in step S3 anddetects the marker 4 from the real image 43 (step S4). The processing ofstep S4 is processing that detects the marker 4 associated with theembroidery pattern arranged in an augmented reality space from the realimage captured by the image sensor 134. The marker 4 is detected and thecoordinates are identified using a well-known method (see JapaneseLaid-Open Patent Publication No. 2010-246885, for example). Morespecifically, feature points are extracted from the real image bypublicly known image processing. The extracted feature points arecompared to feature points extracted from the data indicating the marker4 stored in the marker storage area 137 of the flash memory 124, and thefeature points included in the marker 4 are detected. With regard to thefeature points included in the marker 4, two-dimensional coordinates inan image coordinate system are calculated using the Hough transformprocess, for example. The feature points included in the marker 4 are,for example, end points and intersection points of the character pattern47, and points at four corners of the graphic pattern 48. Then, thetwo-dimensional coordinates in the image coordinate system of thefeature points included in the marker 4 are converted intothree-dimensional coordinates in a world coordinate system. Sinceembroidery coordinate system and the world coordinate system areassociated with each other, the coordinates in the embroidery coordinatesystem are calculated on the basis of the three-dimensional coordinatesin the world coordinate system calculated by image processing.

The CPU 121 determines whether the marker 4 is detected by the imageprocessing of step S4 (step S5). If the marker 4 is not detected (no atstep S5), the CPU 121 then performs the processing of step S20 that willbe described later. If the marker 4 is not detected (no at step S5), theCPU 121 may output a command to display the real image 43 obtained instep S3 to the displaying portion 135. The CPU 121 may output a commandto display a message such as “Please capture marker.” in addition to thereal image 43.

If the marker 4 is detected (yes at step S5), the size and arrangementof the pattern image 46 is determined on the basis of the size andarrangement of the detected marker 4 in the real image 43 (step S6). TheCPU 121 sets, on the basis of the pattern image data received in stepS1, the initial size of the embroidery pattern 5 with respect to themarker 4 such that the size of the embroidery pattern 5 in the Ydirection is 1.7 times the size of the marker 4 in the Y direction. TheCPU 121 sets the initial arrangement of the embroidery pattern 5 withrespect to the marker 4 such that the center of the marker 4 is alignedwith the center of the embroidery pattern 5. The CPU 121 slants theembroidery pattern 5 according to the slant of the marker 4.

The CPU 121 detects an outline of an object (sewing workpiece C) onwhich the marker 4 is arranged, on the basis of the real image 43obtained in step S3 (step S7). The outline of the object (sewingworkpiece C) is obtained by providing the real image 43 to a publiclyknown filter such as a Prewitt filter or a Sobel filter, for example.Specifically, an outline 140 (refer to FIG. 7) of the T-shirt 2 isdetected from the real image 43 by the processing of step S7. The CPU121 may obtain the outline of the object on the basis of data renderedby the user using the touch panel 132. In step S7, if a portion of theobject does not fit the real image, the outline of the portion of theobject that does fit in the real image is detected. That is, there arecases where the outline of the object detected on the basis of the realimage in step S7 does not match the outline of the actual object.

If the embroidery pattern 5 is arranged in the position determined instep S6, the CPU 121 determines whether the pattern image 46 indicatingthe embroidery pattern 5 fits within a closed region surrounded by theoutline 140 detected in step S7 (step S8). The area bounded by theclosed region including the marker 4 is an area where the sewingworkpiece C is extended. The sewing machine 1 is able to sew theembroidery pattern on the sewing workpiece C that extends in a regionsmaller than the sewing area by adhering the sewing workpiece C to afusible interfacing or the like. Therefore, in this example, the regionwhere the embroidery pattern 5 can be arranged is a region in theeditable area, and a portion of the embroidery pattern is permitted tobe arranged outside the closed region (sewing workpiece C) if it is inthe editable area. In this example, if the portion of the embroiderypattern is determined that the pattern image 46 does not fit within theclosed region (no at step S8), the CPU 121 performs deletion processingthat removes the region outside the outline 140 from the pattern image46 (step S10), and outputs a command to display a screen in which theregion outside the outline 140 generated in the deletion processing hasbeen removed from the pattern image 46 on the displaying portion 135(step S11).

If the pattern image 46 fits within the closed region (yes at step S8),the CPU 121 outputs a command to display a pattern image 46 and an areaimage 45 superimposed on the real image 43 to the displaying portion 135(step S9). The displaying portion 135 displays the pattern image 46indicating embroidery pattern 5 associated with the marker 4,superimposed on the real image 43 with the marker 4 detected in step S4as a reference, on the basis of a command from the CPU 121. Thedisplaying portion 135 further displays the area image 45 indicating theeditable area of the embroidery pattern 5, in addition to the patternimage 46, superimposed on the real image 43 with the detected marker 4as a reference, on the basis of a command from the CPU 121. Thedisplaying portion 135 displays a screen 44 and a screen 90 asillustrated in FIG. 7. The screen 44 includes the real image 43, thearea image 45, and the pattern image 46. The real image 43 is an imageobtained in step S3. The area image 45 is an image indicating theeditable area of the embroidery pattern 5. The area image 45 indicatesthe editable area with a dotted rectangle. In the area image 45, theregion inside the dotted line is transparent. The CPU 121 causes thearea image 45 corresponding to the type of the embroidery frame 50indicated by the ID of the embroidery frame 50 to be displayedsuperimposed on the real image 43. The CPU 121 displays the area image45 corresponding to the ID of the embroidery frame 50 obtained in stepS1 at the initial size and arrangement with respect to the marker 4. Thepattern image 46 is displayed at the size and arrangement (position andangle) determined in step S6, on the basis of the pattern image dataobtained in step S1. In the pattern image 46, the region where stitchesare not formed is transparent.

The screen 90 is adjacent to the screen 44 and above the screen 44 inFIG. 7. The screen 90 includes input keys 91 to 94. The input key 91 isa key for inputting a color change command to change thread color of theembroidery pattern 5. When the input key 91 is selected, a color paletteindicating thread colors that are available for the user is displayed,and the user is able to select the color of the embroidery pattern 5from among the colors displayed in the color palette. With an embroiderypattern formed by stitches of a plurality of colors, the thread colormay be changed for each partial pattern of each of the plurality ofcolors. The input key 92 is a key for inputting a command to change thesize of the embroidery pattern 5. When the input key 92 is selected, aninput field for inputting the size in the X direction and the Ydirection of the embroidery pattern 5 is displayed, and the user is ableto input the size of the embroidery pattern in the displayed inputfield.

The input key 93 is a key for inputting a command to rotate theembroidery pattern 5. When the input key 93 is selected, an input fieldfor inputting the rotation angle of the embroidery pattern 5 isdisplayed, and the user is able to input the angle of the embroiderypattern in the displayed input field. It will be assumed that therotation angle is defined such that the clockwise angle in the XY planein the embroidery coordinate system is a positive angle. The input key94 is a key for inputting a command to end the editing of the embroiderypattern 5 represented by the pattern image 46 displayed on thedisplaying portion 135. A command to change the position of theembroidery pattern 5 is input by selecting the position of theembroidery pattern 5 after the move while the pattern image 46 isselected. A command to change the position of the editable area isinputted by selecting the position of the editable area after the movewhile the area image 45 is selected. In this example, the position ofthe editable area with respect to the marker 4 can be changed within arange where the entire marker 4 fits within the editable area.

The CPU 121 determines whether a command to move the editable area hasbeen input (step S12). The CPU 121 determines that the command to movethe editable area has been input (yes at step S12) when the position ofthe editable area after the move has been selected within a range wherethe entire marker 4 fits within the editable area, while the area image45 is selected. That is, the CPU 121 receives a change of the displayposition of the area image 45 within a range where the marker 4 isarranged within the editable area indicated by the area image 45. Whenan operation (1) in FIG. 8, in which a command to move the editable areafrom the state in screen 44 to the right in FIG. 8, is performed (yes atstep S12), the CPU 121 obtains the movement amount of the editable area(step S13). The CPU 121 calculates the movement amount from thedifference between the current position of the editable area and theselected position of the editable area after the move. The movementamount may be indicated in a direction. The direction indicating themovement amount may be a direction based on the upper, lower, left, andright directions of the marker 4. The direction indicating the movementamount may be coordinates from a representative point (for example, thecenter point) of the marker 4. The movement amount may be the currentcoordinates of the editable area calculated from the real image and thecoordinates after the move. When moving the editable area by themovement amount obtained in step S13, the CPU 121 determines whether theembroidery pattern 5 will fit within the editable area (step S16). Ifthe embroidery pattern 5 will fit within the editable area (yes at stepS16), the CPU 121 outputs to the displaying portion 135 a command todisplay a screen that reflects the change of the position of theeditable area with respect to the marker 4, on the basis of the movementamount obtained in step S13 (step S17). Specifically, the CPU 121outputs to the displaying portion 135 a command to display a screen 144on the displaying portion 135 in response to the execution of operation(1), for example. The screen 144 includes the real image 43, the areaimage 45, and the pattern image 46. The area image 45 is arranged to theright compared with the area image 45 of the screen 44, in accordancewith the movement amount obtained in step S13.

The CPU 121 changes area data indicating the position of the editablearea with respect to the marker 4 on the basis of the movement amountobtained in step S13, and stores the changed area data in the flashmemory 124 (step S18). The CPU 121 then returns the processing to stepS3. If the embroidery pattern 5 does not fit within the editable area(no at step S16), the CPU 121 reports an error (step S19). The CPU 121outputs to the displaying portion 135 a command to display “Editing inwhich embroidery pattern protrudes from editable area is invalid.” as anerror message, for example. The CPU 121 discards the movement amountobtained in step S13 and returns the processing to step S3 withoutreflecting the movement of the editable area.

If a command to move the editable area has not been input (no at stepS12), the CPU 121 determines whether a command to edit the embroiderypattern 5 has been received (step S14). In step S14, the CPU 121receives editing with respect to the embroidery pattern 5 via the touchpanel 132. When a command to change the position of the embroiderypattern 5 with respect to the marker 4 has been input, and one of theinput keys 91 to 93 has been selected and an edit command has beeninput, the CPU 121 determines that a command to edit the embroiderypattern 5 has been obtained. If a command to edit the embroidery pattern5 is received (yes at step S14), the CPU 121 obtains a command todisplay the editing content (step S15).

When the input key 91 is selected and an operation (2) in which thecolor red is selected from the color palette is executed while thescreen 144 in FIG. 8 is displayed (yes at step S14), the CPU 121 obtainsa command to change the thread color of the embroidery pattern 5 to red(step S15). When a pattern image 146 is selected and an operation (3) inwhich the position of the embroidery pattern 5 after the move isselected is executed while a screen 244 in FIG. 8 is displayed (yes atstep S14), the CPU 121 obtains a command to move the embroidery pattern5 from the current position to the selected position (step S15). In thiscase, the CPU 121 may obtain the movement amount of the embroiderypattern 5, similar to the processing of step S13. When the input key 93is selected and an operation (4) in which 15 degrees is input for therotation angle is executed while a screen 344 in FIG. 8 is displayed(yes at step S14), the CPU 121 obtains a command to set the rotationangle of the embroidery pattern 5 to 15 degrees (step S15). When theinput key 92 is selected and an operation (5) in which the embroiderypattern 5 is enlarged to 1.7 times the size is executed while a screen444 in FIG. 8 is displayed (yes at step S14), the CPU 121 obtains acommand to enlarge the embroidery pattern 5 to 1.7 times (step S15).

When the pattern image 146 is changed according to the editing contentreceived in step S14, the CPU 121 determines whether the pattern image146 will fit within the editable area indicated by the area image 45displayed on the displaying portion 135 (step S16). If the pattern image146 will not fit within the editable area indicated by the area image(no at step S16), the CPU 121 reports an error (step S19). The CPU 121outputs to the displaying portion 135 a command to display “Editing inwhich embroidery pattern protrudes from editable area is invalid.” as anerror message, for example. The CPU 121 voids the editing contentobtained in step S15 and returns the processing to step S3. When editingwith respect to the embroidery pattern 5 has been received (yes at stepS14), only valid editing content will be reflected in the pattern image146 being displayed on the displaying portion 135. The CPU 121 mayperform processing that enables editing of the embroidery pattern 5within a range where the pattern image 146 fits within the editable areain step S19. For example, when a command, as an editing content, tochange the position of the embroidery pattern 5 is received, the CPU 21may perform processing that enables a change of the position of theembroidery pattern 5 within the range where the pattern image 146representing the embroidery pattern 5 fits within the editable area.That is, the CPU 21 may enable only the editing content within the rangein which the pattern image 146 representing the embroidery pattern 5fits within the editable area. Therefore, the pattern image 146 can onlymove in the editable area.

If it is determined that the embroidery pattern 5 will fit within theeditable area (yes at step S16), the CPU 121 outputs to the displayingportion 135 a command to update the screen on the basis of the editingcontent obtained in step S15 (step S17). In step S17, the editingcontent received in step S14 is reflected in the pattern image 146 beingdisplayed on the displaying portion 135. If the operation (2) describedabove is executed while the screen 144 in FIG. 8 is displayed (yes atstep S14), the displaying portion 135 displays the screen 244 on thebasis of a command from the CPU 121 (step S17). The screen 244 includesthe real image 43, the area image 45, and the pattern image 146. Thepattern image 146 is an image that represents the embroidery pattern 5in which the thread color, i.e., the color of the embroidery pattern 5,has been changed to red in step S15.

When the operation (3) described above is executed while the screen 244in FIG. 8 is displayed (yes at step S14), the displaying portion 135displays the screen 344 on the basis of a command from the CPU 121 (stepS17). The screen 344 includes the real image 43, the area image 45, andthe pattern image 146. The pattern image 146 of the screen 344 isarranged to the lower right by the movement amount obtained in step S15,compared to the pattern image 146 of the screen 244.

When the operation (4) described above is executed while the screen 344in FIG. 8 is displayed (yes at step S14), the displaying portion 135displays a screen 444 on the basis of a command from the CPU 121 (stepS17). The screen 444 includes the real image 43, the area image 45, andthe pattern image 146. The pattern image 146 of the screen 444 isrotated by the rotation amount obtained in step S15, compared to thepattern image 146 of the screen 344.

When the operation (5) described above is executed while the screen 444in FIG. 8 is displayed (yes at step S14), the displaying portion 135displays a screen 544 on the basis of a command from the CPU 121 (stepS7). The screen 544 includes the real image 43, the area image 45, and apattern image 246. The pattern image 246 on the screen 544 is an imagein which the pattern image 146 on the screen 444 has been enlarged by anenlargement factor obtained in step S15.

The CPU 121 changes editing data indicating the size and arrangement ofthe embroidery pattern 5 with respect to the marker 4 as well as thecolor of the embroidery pattern 5, on the basis of the editing contentobtained in step S15, and stores the changed editing data in the flashmemory 124 (step S18). The editing data when the operation (2) describedabove is executed is data that changes the thread color to red. Theediting data when the operation (3) described above is executed is dataindicating the movement amount of the embroidery pattern 5 with respectto the marker 4. The editing data when the operation (4) described aboveis executed is the rotation amount of the embroidery pattern 5 withrespect to the marker 4. The editing data when the operation (5)described above is executed is the enlargement factor of the embroiderypattern 5. After the processing of step S18, the CPU 121 then returnsthe processing to step S3.

In step S6 after the processing has returned to step S3, the CPU 121determines the size and arrangement of the pattern image 246 withrespect to the marker 4, as well as the arrangement of the area image,on the basis of the editing data and the area data stored in the flashmemory 124 in step S18 (step S6). In the screen 544, when the patternimage 246 is displayed superimposed on the real image 43 at the size andarrangement determined in step S6, the pattern image 246 will not bearranged in the closed region surrounded by the outline 140 detected instep S7 (no at step S8). In such a case, the CPU 121 outputs to thedisplaying portion 135 a command to display an image in which theportion outside the closed region surrounded by the outline 140 detectedin step S7 has been deleted from the pattern image (step S11). Thedisplaying portion 135 displays a screen 644 illustrated in FIG. 9 inthe portion where the screen 44 in FIG. 7 is displayed, in response to acommand from the CPU 121. The screen 644 includes the real image 43, thearea image 45, and a pattern image 346. The pattern image 346 is animage in which a portion 141 outside the closed region surrounded by theoutline 140 detected in step S7 has been deleted from the pattern image246 in the screen 544. After step S11, the CPU 121 performs the sameprocessing of step S12 described above.

If a command to edit the embroidery pattern 5 is not obtained (no atstep S14), the CPU 121 determines whether the input key 94 is selected(step S20). If the input key 94 is not selected (no at step S20), theCPU 121 returns the processing to step S3. If the input key 94 isselected (yes at step S20), the CPU 121 performs processing thatdisplays editing data indicating the editing content obtained in stepS15 (step S21). Specifically, the CPU 121 outputs to the displayingportion 135 a command to display a screen indicating the editing dataand the area data stored in step S18 (step S21). When the operations (1)to (5) are executed, the CPU 121 outputs to the displaying portion 135 acommand to display the color, movement amount, rotation amount, andenlargement factor of the embroidery pattern 5 as the editing data. Thatis, the editing data includes position information and size information.The position information is information indicating the position of theembroidery pattern 5 with the position of the marker 4 as a reference.The size information is information identifying the size of theembroidery pattern 5. The CPU 121 displays the movement amount of thesewing area with respect to the marker 4 as area data. The CPU 121 sendsthe editing data and the area data to the sewing machine 1 via thecommunication I/F 125 (step S22). Then the CPU 121 ends the mainprocessing. The CPU 121 may end the main processing when notificationthat reception is complete is received from the sewing machine 1, withrespect to the data sent in step S22.

[8. Processing Performed by Sewing Machine 1]

The main processing performed by the sewing machine 1 will be describedwith reference to FIG. 10. The main processing of the sewing machine 1is performed when a start command is input by a user after theembroidery pattern to be sewn and the embroidery frame 50 to be usedhave been selected. In the main processing, processing that arranges theembroidery pattern with respect to the arrangement of the marker 4captured by the sewing machine 1, in accordance with the content editedby the device 3, and sewing the embroidery pattern is performed. When acommand is detected, the CPU 61 of the sewing machine 1 reads a programfor performing the main processing stored in the program storage area ofthe ROM 62 to the RAM 63. The CPU 61 performs the following stepsaccording to commands included in the program read to the RAM 63.Various parameters necessary for performing the main processing arestored in the flash memory 64. Various data obtained throughout the mainprocessing is stored in the RAM 63 as appropriate. Hereinafter, in orderto simplify the description, a case will be described in which theembroidery pattern 5 to be sewn is selected and the embroidery frame 50to be used when sewing the embroidery pattern 5 is selected in thesewing machine 1, and the embroidery pattern 5 is edited in accordancewith the main processing of the device 3, in the device 3, before themain processing starts.

As illustrated in FIG. 10, the CPU 61 of the sewing machine 1 sends thedata indicating the embroidery pattern to be sewn and data indicatingthe editable area to the device 3 via the communication I/F 67 (stepS41). The data indicating the editable area is data indicating the typeof the embroidery frame 50. In step S41, the pattern image data and theID of the embroidery frame 50 are sent to the device 3. The CPU 61determines whether editing data has been received from the device 3(step S42). The editing data of step S42 is editing data to be sent fromthe device 3 in step S22 of FIG. 5. If the editing data has not beenreceived (no at step S42), the CPU 61 waits until the editing data isreceived. If the editing data has been received (yes at step S42), theCPU 61 corrects embroidery data of the embroidery pattern to be sewn, onthe basis of the editing data received in step S42 (step S43).Specifically, the color thread data of the sewing data is changed to redon the basis of the editing data. The coordinate data of the sewing datais corrected on the basis of the rotation amount and the enlargementfactor of the embroidery pattern indicated by the editing data.

The CPU 61 reports the attaching position indicating the marker 4 withrespect to the selected embroidery frame 50 to be used (step S44). TheCPU 61 determines the attaching position on the basis of the position ofthe editable area with respect to the marker 4 indicated by the areadata received in step S42, and the position of the sewing area 54 withrespect to the embroidery frame 50 stored in the sewing area storagearea 70. The CPU 61 displays a screen 97 in FIG. 10, and displays theposition of the marker 4 with respect to the embroidery frame 50, on theLCD 15, for example. The sewing area 54 may be displayed on the screen97, but it is also acceptable not to display the sewing area 54 on thescreen 97. The position of the sewing area 54 with respect to the marker4 on the screen 97 is the same as the position of the editable area withrespect to the marker 4 on the screen 644 in FIG. 9. The user causes theembroidery frame 50 to hold the T-shirt 2 while referring to the LCD 15.The marker 4 captured by the device 3 when editing the embroiderypattern 5 is affixed to the T-shirt 2 without being peeled off. The userattaches the embroidery frame 50 holding the T-shirt 2 to the sewingmachine 1. The CPU 61 determines whether the embroidery frame 50 isattached on the basis of an outputted signal from the detector 36 (stepS45). If the embroidery frame 50 is not detected (no at step S45), theCPU 61 waits until the embroidery frame 50 is detected.

If the embroidery frame 50 is detected (yes at step S45), the CPU 61obtains the type of embroidery frame and identifies the ID of theembroidery frame 50 on the basis of an outputted signal from thedetector 36 (step S46). The CPU 61 may output to the displaying portion135 a command to display an error message if the ID of the embroideryframe 50 identified in step S46 is different from the ID of theembroidery frame 50 sent in step S41.

The CPU 61 outputs a capture command to the image sensor 35 (step S47).The image sensor 35 starts processing that captures an image within acapture area and outputs image data to the CPU 61 on the basis of acommand from the CPU 61. The CPU 61 obtains the image data outputtedfrom the image sensor 35 (step S48). The CPU 61 may obtain image dataindicating the entire sewing area, or may obtain image data indicating aportion, within the sewing area 54, that includes the attaching positionof the marker 4 with respect to the sewing area 54 reported in step S44.The CPU 61 image-processes the image data obtained in step S48 anddetects the marker 4 included in the image by processing similar to theprocessing of step S4 in FIG. 5 (step S49).

The CPU 61 determines whether the marker 4 is detected with step S49(step S50). If the marker 4 is not detected (no at step S50), the CPU 61returns the processing to step S44. The CPU 61 may display an errormessage such as “Marker not detected. Please reattach sewing workpieceC.” If the marker 4 is detected (yes at step S50), the CPU 61 arrangesthe embroidery pattern 5 associated with the marker 4, in accordancewith the marker 4 (step S51). The CPU 61 sets the position of theembroidery pattern 5 with respect to the marker 4 to a position moved bythe movement amount included in the editing data from the initialposition. The CPU 61 slants the embroidery pattern 5 according to theslant of the marker 4.

When the embroidery pattern is arranged according to the marker 4, theCPU 61 determines whether the embroidery pattern fits within the sewingarea 54 (step S52). The sewing area 54 is identified on the basis of thetype of the embroidery frame 50 obtained in step S46, and thecorrespondence between the type of the embroidery frame 50 stored in thesewing area storage area 70 and the sewing area. If the embroiderypattern 5 does not fit within the sewing area 54 (no at step S52), theCPU 61 returns the processing to step S44. If the embroidery pattern 5fits within the sewing area 54 (yes at step S52), the CPU 61 determineswhether a command to start sewing has been input by the start/stopswitch 29 being pressed down (step S53). If the command to start sewinghas not been input (no at step S53), the CPU 61 waits until the commandto start sewing is input. If the command to start sewing has been input(yes at step S53), the CPU 61 sews the embroidery pattern in accordancewith the arrangement set in step S51 (step S54). More specifically, theCPU 61 drives the X-axis motor 83 and the Y-axis motor 84 (refer to FIG.3) according to the embroidery data corrected in step S43 and moves theembroidery frame 50. The CPU 61 sews the embroidery pattern 5 onto thesewing workpiece C (T-shirt 2) held by the embroidery frame 50, bydriving the needle bar up-and-down movement mechanism 34 by the sewingmachine motor 81 in conjunction with the movement of the embroideryframe 50 so as to move the needle bar 6 to which the sewing needle 7 isattached up and down. When the sewing of the embroidery pattern 5 isfinished, the CPU 61 ends the main processing as described above. Whenthe embroidery pattern 5 is sewn in accordance with the editing contentof the screen 644 illustrated in FIG. 9, the embroidery pattern 5 may bea pattern in which the portion 141 has been removed, or a pattern thatincludes the portion 141. Specifically, the sewing workpiece C of aspecific example is the T-shirt 2, and the position of an end portion (aportion of a hoop) can be changed so that the entire embroidery pattern5 can be sewn when the sewing workpiece C is held in the embroideryframe 50. The sewing machine 1 or the device 3 may allow the user toselect whether to sew the entire embroidery pattern 5.

The device 3 detects the marker 4 associated with the embroidery patternarranged within the augmented reality space, from the real image 43 thatis an image of the real space captured by the image sensor 134 (stepS4). When the marker 4 is detected from the real image 43 (yes at stepS5), the pattern image 46 indicating the embroidery pattern 5 associatedwith the detected marker 4 is displayed superimposed on the real image43 with the detected marker 4 as a reference, on the displaying portion135 (step S9). The editing content with respect to the embroiderypattern 5 is received via the touch panel 132 (step S12 and step S14).When the editing content with respect to the embroidery pattern has beenreceived (yes at step S12 and yes at step S14), the received editingcontent is reflected in the pattern image 46 displayed on the displayingportion 135 (step S17).

The user is then able to confirm the finished image of the embroiderypattern when the embroidery pattern 5 has been arranged with respect tothe sewing workpiece C and the embroidery pattern 5 has been edited, byreferring to the displaying portion 135. The user can imagine thefinished embroidery pattern 5 by referring to the real image 43 and thepattern image 46. The user can edit the embroidery pattern 5 whileconfirming the finished embroidery pattern 5.

The device 3 outputs the editing data indicating the received editingcontent with respect to the embroidery pattern 5 (step S22). The usercan sew the embroidery pattern 5 with the sewing machine 1 according tothe editing content received by the device 3, on the basis of theoutputted editing data. The device 3 outputs the editing data to thesewing machine 1 via the communication I/F 125 provided in the device 3,so the user is saved the trouble of inputting the editing data to thesewing machine 1. It is thus possible to avoid erroneous entry of theediting data by the user, so the user can sew the embroidery pattern 5with the sewing machine 1 according to the editing content in the device3.

The device 3 receives data indicating the embroidery pattern 5associated with the marker 4 from the sewing machine 1 via thecommunication I/F 125 (step S1). The device 3 causes the pattern image46 to be displayed on the displaying portion 135 on the basis of thedata received from the sewing machine 1 (step S9). The device 3 does notneed to store the data indicating the embroidery pattern 5.

The editing data includes the position information indicating theposition of the embroidery pattern 5 with the position of the marker 4as a reference. The user can change the position of the embroiderypattern 5 with the position of the marker 4 as a reference whileconfirming the finished embroidery pattern 5. The editing data includesthe size information identifying the size of the embroidery pattern 5.The user can change the size of the embroidery pattern 5 whileconfirming the finished embroidery pattern 5.

When the editing content with respect to the embroidery pattern 5 isreceived, the device 3 displays the received editing content on thedisplaying portion 135 (step S21). The user can confirm the receivedediting content by referring to the displaying portion 135. The user cansew the embroidery pattern 5 with the sewing machine 1 according to theediting content in the device 3, by inputting the editing contentdisplayed on the displaying portion 135 to the sewing machine 1.

When the marker 4 is detected from the real image 43 (yes at step S5),the device 3 displays the area image 45 indicating the editable area ofthe embroidery pattern 5, in addition to the pattern image 46,superimposed on the real image 43 with the detected marker 4 as areference. The user can determine, with the displaying portion 135,whether the embroidery pattern 5 is arranged within the editable area.It is possible to suppress the embroidery pattern 5 from being edited ina range beyond the editable area. The user can thus determine whether itis possible to sew the embroidery pattern 5 with the sewing machine 1according to the editing content in the device 3.

The device 3 identifies at least one of the type and the size of theembroidery frame 50 from the sewing machine 1 via the communication I/F125 (step S1). The device 3 displays the area image 45 corresponding tothe at least one of the identified type and the identified size ofembroidery frame 50, in addition to the pattern image 46, superimposedon the real image 43. The device 3 can display the area image 45corresponding to the type or size of the embroidery frame 50 attached tothe sewing machine 1, superimposed on the real image 43. The user cangrasp the editable area according to the type or size of the embroideryframe 50 to be attached to the sewing machine 1, without having to inputthe type or size of the embroidery frame 50 to the device 3.

When the received editing content is reflected in the pattern image 146,the device 3 determines whether the reflected pattern image 146 exceedsthe editable area indicated by the area image 45 displayed on thedisplaying portion 135 (step S8). In response to determining that thepattern image 146 fits within the editable area, an error is displayedon the displaying portion 135 (step S19) or editing of the embroiderypattern 5 within a range where the pattern image 146 is within theeditable area is enabled. When the editing content with respect to theembroidery pattern 5 is received, enabled editing content is reflectedin the pattern image 146 displayed on the displaying portion 135 (stepS18). The device 3 is able to reliably suppress the embroidery pattern 5from being edited in a range beyond the editable area.

The change of the display position of the area image 45 is receivedwithin a range where the marker 4 is arranged within the editable areaindicated by the area image 45 (step S12). When the change of thedisplay position of the area image 45 is received, the device 3 reflectsthe change of the display position (step S17). The user can change theeditable area with respect to the marker 4 within a range in which themarker 4 is arranged within the editable area. The device 3 outputs areadata indicating the position of the area image 45 with respect to theposition of the marker 4 (step S22). The user can obtain the position ofthe area image 45 with respect to the position of the marker 4.

The device 3 detects the outline 140 of the object on which the marker 4is arranged from the real image 43 (step S7). The device 3 determineswhether the pattern image 246 fits within the closed region surroundedby the detected outline 140 (step S8). In response to determining thatthe pattern image 246 does not fit within the closed region, the regionoutside the closed region of the outline 140 is excluded from thepattern image 46 and this is reflected in the pattern image 346displayed on the displaying portion 135 (step S11). The user canappropriately grasp the finished embroidery pattern 5 when a portion ofthe embroidery pattern 5 is arranged beyond the outline 140 of theobject.

The embroidery pattern editing program and the embroidery patterndisplaying device of the present disclosure are not limited to theembodiment described above. Various modifications may be made withoutdeparting from the scope of the present disclosure. For example,modifications (A) to (C) below may be made as appropriate.

(A) The device 3 need only include the image sensor 134, the displayingportion 135, the touch panel 132, and the CPU 121, and the configurationof the device may be modified as appropriate. Each of the image sensor134, the displaying portion 135, and the touch panel 132 may be changed,as appropriate, to publicly known technology. For example, the touchpanel 132 may be the operating switch 131. For example, thecommunication I/F 125 may be omitted from the device 3. The device 3 maybe a mobile terminal such as a smartphone provided with a communicationportion, or the device 3 may be a personal computer. The sewing machine1 need only be able to sew the embroidery pattern 5, and theconfiguration of the sewing machine 1 may be modified as appropriate.The sewing machine 1 may be an industrial sewing machine and amulti-needle sewing machine. The movement mechanism 40 that is themoving portion of the sewing machine 1 need only be able to move thesewing workpiece C in a first direction relative to the image sensor134, and a second direction intersecting the first direction. The movingportion may be a feed dog capable of moving the sewing workpiece C inthe front-rear direction and the left-right direction, for example. Themoving direction (first direction and second direction) of the sewingworkpiece C by the moving portion may be modified as appropriate. Theprocessing performed by the sewing machine 1 may be modified asappropriate.

(B) The embroidery pattern editing program that includes the command forperforming the main processing (refer to FIG. 5) of the device 3 needonly be stored in a storage device of the device 3 until the CPU 121executes the program. Therefore, each of the program acquisition method,the program acquisition path, and the device that stores the program maybe modified as appropriate. The program executed by the CPU 121 may bereceived from another device via a cable or wireless communication andstored in a storage device such as flash memory. The other deviceincludes a PC and a server connected via a network, for example.

(C) The steps of the main processing (refer to FIG. 5) of the device 3are not limited to the example in which the steps of the main processingare performed by the CPU 121. A portion or all of the steps may beperformed by another electronic device (for example, ASIC). The steps ofthe main processing may be distribution-processed by a plurality ofelectronic devices (for example, a plurality of CPUs). The order of thesteps of the main processing can be changed as necessary, and steps ofthe main processing can be omitted and added as necessary. A mode inwhich an operating system (OS) or the like that operates on the device 3performs a portion or all of the main processing on the basis of acommand from the CPU 121 is also included in the scope of the presentdisclosure. For example, modifications (C-1) to (C-6) below may be made,as appropriate, to the main processing.

(C-1) The type of data indicating the embroidery pattern 5 obtained instep S and the method of obtainment may be modified as appropriate. Forexample, the device 3 may store the embroidery data in the flash memory124 or the like. In this case, the CPU 121 may obtain the ID of theembroidery pattern 5 from the sewing machine 1 as the data indicatingthe embroidery pattern 5. There is no need for the sewing machine 1 tobe started at the time of starting the main processing of the device 3.The CPU 121 may obtain the sewing data of the embroidery pattern 5 asthe data indicating the embroidery pattern 5. In this case, the CPU 121may generate the pattern image data on the basis of the sewing data.

The CPU 121 may obtain the data indicating the embroidery pattern 5 viaa communication portion or via a storage device such as a memory card.The CPU 121 may obtain the ID and the like of the embroidery pattern 5input by the user as the data indicating the embroidery pattern 5.

(C-2) The processing that displays the area image indicating theeditable area according to the arrangement of the marker 4 in the realimage may be omitted as appropriate. If the processing that displays thearea image is omitted, the device 3 does not need to send informationidentifying the type of the embroidery frame 50. In a case where theprocessing that displays the area image is performed, the CPU 121 mayobtain data indicating the editable area by obtaining area image dataindicating the area image. The CPU 121 may obtain the data indicatingthe editable area by obtaining information identifying the size of theeditable area. In the CPU 121, the method for obtaining the dataindicating the editable area may be modified as appropriate. The CPU 121may obtain the ID and the like of the embroidery frame 50 input by theuser as the data indicating the editable area. The CPU 121 does not haveto receive a command to change the position of the editable area withrespect to the marker 4 in a real image. The device 3 does not need tostore the initial arrangement of the editable area with respect to themarker 4 for each type of embroidery frame 50. In this case, the device3 may match up a representative point of the marker 4 and arepresentative point of the editable area regardless of the type of theembroidery frame 50, for example. The representative point in thisexample refers to a center point, a vertex, an intersection point, or abending point or the like, for example.

In this example, an area of the same size and shape as the sewing areaset according to the type of the embroidery frame 50 is set as theeditable area, but the present disclosure is not limited to this. Forexample, a sewing area common to a plurality of types of embroiderypatterns may be set as the editable area. In this way, the editable areacan be set without identifying the type of the embroidery frame 50. Theeditable area may be a different size and shape than an area that is thesame size and shape as the sewing area.

(C-3) The editable item of the embroidery pattern 5 in the device 3 maybe modified as appropriate. For example, the CPU 121 may be capable ofediting one of the position, angle, and size of the embroidery patternwith respect to marker 4. The CPU 121 does not have to receive a changeof thread color. The editing data may be changed according to thecontent of editing the embroidery pattern. The editing data does nothave to include position information. The editing data does not have toinclude size information. The device 3 may correct the sewing dataaccording to the editing content, and may send the sewing data after thecorrection to the sewing machine 1 as the editing data in step S22. Themethod of inputting the editing content of the embroidery pattern may bemodified as appropriate.

(C-4) The method of outputting the editing data and the area data may bemodified as appropriate, and the processing that outputs at least one ofthe editing data and the area data may be omitted as necessary. When thedevice 3 includes a speaker, at least one of the editing data and thearea data may be audio output via the speaker. For example, the CPU 121may omit the processing of step S22. In this case, the user can sew theembroidery pattern onto the sewing workpiece C according to the editingcontent in the device 3, by inputting the editing data and the area datadisplayed on the displaying portion 135 in step S21 to the sewingmachine 1.

(C-5) The device 3 may be capable of identifying each of a plurality ofmarkers. In this case, the device 3 may store different embroiderypatterns associated with each marker, and the CPU 121 may identify anembroidery pattern to be sewn, on the basis of the type of markerdetected. In this case, the device 3 may store the type of marker andthe pattern image data of the embroidery pattern in association witheach other, and may display the pattern image on the basis of thepattern image data corresponding to the type of marker. In this way, thedevice 3 does not need to obtain data for displaying the pattern imagefrom the sewing machine 1. When the device 3 stores the pattern imagedata of the embroidery pattern, selection of the embroidery pattern tobe sewn may be performed by the device 3.

(C-6) The CPU 121 may omit the processing of step S7, step S8, step S10,and step S11. The CPU 121 may receive movement within the editable areain the closed region surrounded by the outline detected in step S7. Thedevice 3 may be capable of performing the correction of the embroiderydata performed by the sewing machine 1 in step S43. The CPU 121 may omitthe processing of step S19. The CPU 121 may perform processing thatenables editing of the embroidery pattern within a range where theembroidery image indicating the embroidery pattern is within theeditable area.

The apparatus and methods described above with reference to the variousembodiments are merely examples. It goes without saying that they arenot confined to the depicted embodiments. While various features havebeen described in conjunction with the examples outlined above, variousalternatives, modifications, variations, and/or improvements of thosefeatures and/or examples may be possible. Accordingly, the examples, asset forth above, are intended to be illustrative. Various changes may bemade without departing from the broad spirit and scope of the underlyingprinciples.

What is claimed is:
 1. A non-transitory computer-readable medium storingcomputer-readable instructions that, when executed, cause a processor ofan embroidery pattern displaying device provided with a capturingportion and a displaying portion to perform steps comprising: detectinga marker associated with an embroidery pattern to be arranged within thedisplaying portion as a pattern image, from a real image that is animage of a real-world space captured by the capturing portion;determining, when the marker is detected from the real image, a size ofthe pattern image based on a size of the detected marker in the realimage, the pattern image indicating the embroidery pattern associatedwith the detected marker; and displaying, when the marker is detectedfrom the real image, the pattern image having the determined sizesuperimposed on the real image with the detected marker as a reference,on the displaying portion, a relative position of the pattern imagebeing predetermined with respect to a position of the marker, thepattern image being automatically aligned with the marker in therelative position of the pattern image.
 2. The non-transitorycomputer-readable medium according to claim 1, wherein thecomputer-readable instructions further cause the processor to performsteps comprising: receiving a predetermined input via an input portion;and reflecting, when the predetermined input is received, the receivedpredetermined input in the pattern image displayed on the displayingportion, the received predetermined input including changing a size ofthe pattern image or changing a position of the pattern image, on thebasis of the detected marker as a reference.
 3. The non-transitorycomputer-readable medium according to claim 2, wherein: the receivingthe predetermined input includes receiving an input of an editingcontent with respect to the embroidery pattern, and the reflecting stepincludes reflecting, when the input of the editing content with respectto the embroidery pattern, the received editing content in theembroidery pattern displayed on the displaying portion, the receivedediting content including changing a size of the embroidery pattern orarrangement of the embroidery pattern.
 4. The non-transitorycomputer-readable medium according to claim 3, wherein thecomputer-readable instructions further cause the processor to perform astep comprising: outputting editing data indicating the received editingcontent.
 5. The non-transitory computer-readable medium according toclaim 4, wherein the outputting the editing data includes outputting theediting data to a sewing machine via a communication portion provided inthe embroidery pattern displaying device.
 6. The non-transitorycomputer-readable medium according to claim 4, wherein the editing dataincludes position information indicating the position of the embroiderypattern with the position of the marker as the reference.
 7. Thenon-transitory computer-readable medium according to claim 4, whereinthe editing data includes size information specifying a size of theembroidery pattern.
 8. The non-transitory computer-readable mediumaccording to claim 4, wherein the outputting the editing data includes,when the editing content with respect to the embroidery pattern isreceived, displaying the received editing content on the displayingportion.
 9. The non-transitory computer-readable medium according toclaim 3, wherein the displaying on the displaying portion includes, whenthe marker is detected, displaying an area image indicating an editablearea of the embroidery pattern, in addition to the pattern image,superimposed on the real image with the detected marker as thereference.
 10. The non-transitory computer-readable medium according toclaim 9, wherein the computer-readable instructions further cause theprocessor to perform a step comprising: receiving informationidentifying at least one of a type and size of an embroidery frame froma sewing machine via a communication portion provided in the embroiderypattern displaying device, wherein the displaying on the displayingportion includes displaying the area image corresponding to the at leastone of the identified type and the identified size of the embroideryframe, in addition to the pattern image, superimposed on the real image.11. The non-transitory computer-readable medium according to claim 9,wherein the computer-readable instructions further cause the processorto perform steps comprising: determining whether the pattern imagedisplayed on the displaying portion fits within the editable areaindicated by the area image displayed on the displaying portion; andperforming processing that enables only editing content of theembroidery pattern within a range in which the pattern image fits withinthe editable area, wherein the reflecting includes, when the editingcontent with respect to the embroidery pattern is received, reflectingthe editing content enabled by the processing in the pattern imagedisplayed on the displaying portion.
 12. The non-transitorycomputer-readable medium according to claim 9, wherein thecomputer-readable instructions further cause the processor to perform astep comprising: receiving a change of a display position of the areaimage, within a range in which the marker is arranged within theeditable area indicated by the area image, wherein the reflectingincludes, when the change of the display position of the area image isreceived, reflecting the change of the display position.
 13. Thenon-transitory computer-readable medium according to claim 9, whereinthe computer-readable instructions further cause the processor toperform a steps comprising: outputting area data indicating a positionof the area image with respect to a position of the marker.
 14. Thenon-transitory computer-readable medium according to claim 1, whereinthe computer-readable instructions further cause the processor toperform a step comprising: receiving data indicating the embroiderypattern associated with the marker from a sewing machine via acommunication portion provided in the embroidery pattern displayingdevice, wherein the displaying on the displaying portion includesdisplaying the pattern image on the displaying portion on the basis ofthe received data from the sewing machine.
 15. The non-transitorycomputer-readable medium according to claim 1, wherein the displayingincludes, when the marker is detected, displaying a key to receive anediting content with respect to the embroidery pattern in addition tothe embroidery image.
 16. The non-transitory computer-readable mediumaccording to claim 15, wherein the key includes a key to receive acommand that changes a size of the embroidery pattern.
 17. Thenon-transitory computer-readable medium according to claim 15, whereinthe key includes a key to receive a command that changes an angle of theembroidery pattern.
 18. The non-transitory computer-readable mediumaccording to claim 17, wherein the computer-readable instructionsfurther cause the processor to perform steps comprising: detecting anoutline of an object on which the marker is arranged from the realimage; and determining whether the pattern image fits within a closedregion surrounded by the detected outline, wherein the displayingincludes, in response to determining that the pattern image does not fitwithin the closed region, displaying the pattern image in which a partof the pattern image that is outside the closed region of the outline isexcluded.
 19. An embroidery pattern displaying device comprising: acapturing sensor configured to capture a real-world image that is animage of a real space of an object; a display configured to display thereal-world image captured by the capturing sensor, and a pattern imageindicating an embroidery pattern; a processor; and a memory storingcomputer-readable instructions that, when executed by the processor,cause the embroidery pattern displaying device to: detect a markerassociated with the embroidery pattern to be arranged within the displayas a pattern image, from the real-world image captured by the capturingsensor; determine, when the marker is detected from the real image, asize of the pattern image based on a size of the detected marker in thereal image, the pattern image indicating the embroidery patternassociated with the detected marker; and display, when the marker isdetected, the pattern image having the determined size superimposed onthe real-world image captured by the capturing sensor with the detectedmarker as a reference, on the display, a relative position of thepattern image being predetermined with respect to a position of themarker, the pattern image being automatically aligned with the marker inthe relative position of the pattern image.
 20. A method of controllingan embroidery pattern displaying device provide with a capturing portionand a displaying portion, the method comprising steps of: detecting amarker associated with an embroidery pattern to be arranged within thedisplaying portion as a pattern image, from a real image that is areal-world image of a real-world space captured by the capturingportion; determining, when the marker is detected from the real image, asize of the pattern image based on a size of the detected marker in thereal image, the pattern image indicating the embroidery patternassociated with the detected marker; and displaying, when the marker isdetected, the pattern image having the determined size superimposed onthe real image captured by the capturing portion with the detectedmarker as a reference, on the displaying portion, a relative position ofthe pattern image being predetermined with respect to a position of themarker, the pattern image being automatically aligned with the marker inthe relative position of the pattern image.